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ABSTRACT - ♦ • - 

^ " ' The validity of predictiot equat'ions based on , / 

American College Testing (ACTV Program test, scores ind high'schpoi' 
grades is inve'sti gated'. Presented are sepaAte results for prediction 
eguations bisdd on ACT scojres only, on high school grades cnly, and 
on both kinds of^predjicticn icintly. Also included is a discussion of 
the relationship between equations and the Jl'ength of time th^^r^have 
fieeft in use. 'The stud y^ is based on data collected from a random^ 
sample of ^260 colleges participating in the ACT Research Services 
from 1972<r 73 tb *1 976- 77 . The accuracy off prediction eguations based 
on the two variables was found. to be guite stable over tiie. A ' \ 
similar stability was rioted In the accuracy of- grade . predictions for ' 
males and females separately. Grade predictions -^based cn ACT scores 
only sho we'd slightly greater ^tabili^ty over time thaa those baseid on 
grades only. It is concluded that despite the changes' some colleges 
experience in student abilities^ curricula, and gradiig practices^ 
over time, ^ freshman grade averages can be accurately 'predicted using 
equations as old, as f9ur years. ACT pclicyis to update equations at 
least* every three yearsv Dlata tibles and statistical analyses are 
briefly discussed, ?ind a brief list of references is included. 
(Author/MSE) / < \. > 



■;*'*>, Beproductions supplied by EDRS.are the best that ca^ -be made • 
'* . ^ , from the original document, \ . * 



% 



( 



/ 



Prepared, by the Research and Development Division 
The American College Testing Program 

For adciitTOnal copies write: ■ 
ACT Publications * ^ 
P.O. Box 168 , 

Iowa City, Iowa 52243 . " ^ 

^ ♦ • ' . ■ 

.(Check or money order must accompany request.) Price: $1:00 ; 



ERIC 



^'1979 by The American College Testing Program. All rights reserved. 



\ 



4 



ABSTRACT ; , 

This report investigates the validity, of prediction equations based on ACT test scores.ar>d high 
school grades. It presents separate results for prediction equations based, on ACT test scores 
<:inly. on high schooPgradds only, and on both kinds of predictio^l jointly. It Slso/ipcludes a 
discussign of the relationship between the validity of tJie prediction equations and the length of 
trme they have beea in use, ♦ i • ^ % *^ * 

. ■ ■ ■ / . A r ■ 

The study is based on daja collected from a random sample of 260 colleges tnat participated rfi 
the ACT Research Servic^s from 1972-73 to 1976-7/'. Separate. prediction equations for each 
college were calculated 'from data for the years.1972-73, 1973-74, 1974-75, and 1^75-76. These 
prediction" equations were then applied to data from 1976-77 freshmen, and tfvp predicted and 
actual grade^ were compared. * ^ ^ , ' If ' 

*■ < * ■ 

The accuracy 0/ the prediction equations based on ACT test scores ahd-h^h school grades 
jointly was quite stable over time. The meah absolute error of prediction rar^eid from ,534o .55 
over the four years: the proportion of stixients whose predicted grade was' within ,20 of their 
earned grade ranged from ,24 to ,25; and the cross-validated correlation ranged from .55 to*,56, 
A similar stability was noted ip the accuracy of grade predictions fot males and females 
separately, • • ^ - 

Grade predictions based only on ACT te&t^ores had a slightly smalfer mean absolute Bfror 
than grade predictions based only on hi^n school grades, and showed slightly. greater stabfl|.ty 
over time. The rnedn absolute error of predictions based only on test-scores Was about ,57 for 
the whole four-year period,. fOr^fSfedictions based only on higf^^'schoql grades, the mean 
absolute error ranged from ,57xfo ,60 over the four-year period, . 

One can conclude that although most colleges experience some change in their'shadents' 
academic abilities. <he|jr curricula, or their grading practices. Over a period of time, freshman 
grade average can ip-most c^es be accurately predicted using equations as old as four years. It 
IS ACT policy that colleges ifpdate their prediction eq^uations at least every three years if they, 
are to receive grade- predictions for future applicants, Th0 data from this study suggest that 
• more frequerft revision is not generally necessary. In .«onaicleringwhether revision of equations 
. rs necessary beforQ the required date, college researttherg^should, of course,* determine whether 
some change has occurred which would necessitate.^ early revision. - • - 

^ ■ * ■ r ■■ . ■ '■ \ 



THE VALIDITY OVER TIME OF COLLEGE FRESHMAN GRADE PREDICTfON EQUATIONS 



•A problem' commonly^ encouotefed in predicling 
college freshman grades, from standardized test 
scores and hi^ school, grades is the validity over 
time of the: prediction Afuations. Changes over time 
•in the di^ribution of, ability among entering sty- 
dents, in a.colle*ge's entrance requirements, in its 
freshman.,^curricjulum, and iri.jr)sl;fuct(?rs' grading 
policies l^n be 'great enough to m^ke old-predrc- 
tion -eqtiations inaccurate. Deterioration • of 
tfie accuracy of gr^de- j/re'dictions. has' obvi'ocfe 
negativ6. impJicationS/.fOr ,bc«h. cblT^g^s ahd 
students. On the 'et'hei* hand,' collectirig 'and 
r^portmg the data needed to revise pretfiction 
equations can -be expensive, and- trme-cpnsuming. 
Therefore, many coNege researeheVs 'W<5uld 
naturally want . to use pre;diction equjations as long 
a% possible before uadating them. 

Td'Jaccpmmodate the tiri^e schedules of colleges^, 
AC'J updates its prediction equa<ions eacb fall 
usmg data' collected fromv students who were 
freshmen in the previous year. Predictlohequ^ations 
•a?e, therefore, based on freshman^jfjrades-that are at 
(east one year old. BWause most students who take 
tine"" ACT Assessfrnent ^a/all"of one Vear will be 
x^ollege fre^lVnen in thV'^all of t|ie next -year. 
•:predictiorrw,eigh|ji .are. typically calculated from 
college grades- that are at least two yBars oldeHhan 
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the grades being predicted. To minimize the error 
from out-of-date equations. ACT requires colleges 
to participate^in its predictive research sefVices at 
least once every three years if they wish to continue 
to receive grade predictions for future applicants., 
Thus, predicted grades. are typically based on data', 
that ar^ two to four years older than the grades 
being predicted. 

The primary purpose of thiis report isr to document 
the relationship between the validity and the age of 
grade prediction equations based on ACT Assess- 
'menftest scores and high school grades. A second 
punpose. is to summarize the validity of locally 
developed grade prediction equations" based on 
ACT test scores only" on high school grades only, 
and on both kinds' of predictors jointly. A third" 
p urpose is to suggest yaripus techniques for 
'\ eva<uaTm5"1«G^l^rediction equations. 

Prediction equations were calculated for a 
probability sample of individual colleges using data 
from -the, years 1972-73; 1973-74. 1974-75. apd 
1975-76. The predicted grades from these, 
"equations were then compared' with the actual 
freshman grades earned in 1976-77. and the • 
comparisons are stated in terms that are easily 
int^preted and used. 



Previous Research 



There is relatively Jittje p^ublished research on the 
rate* at. which predictive accuracy , declines over 
uMe. Hfc{!s.*KlQck,^and^Bush .(1965j corripared t . 
.pi'pdicteJand earrted'^ades pi. students at seven 
cpllepes in Georgia over a4hree-year period. They 
f^und tha;i;'-t3to^ co.rrelatioft between the 

prep^Xed he]^^i^ grade ^average and the actual 
grade Average orie and tWo^e&rs ^ater was .64 and^ 
63. respectively. The average correlation between 
trBSltyi^an gradeiivergge and'predicto/s in the base 
year t ^ ' 4 

BQwers anrf kipeb (1972Mound thaf in predicting 
graW$s for freshmen at th6 University qf lllinors, the 



weights for ACT Composite scores were unstable 
over a five-year periodr They.found that the weight 
for 'high school percentile rank as a predictor of^ 
• freshman grades was'more stable. 



Perrin and VVhitney (1976) studied the ACT scores^ 
high school grades, and freshman grades ffom a 
national sample .of- studen; records. Tbey found 
very little difference in the accuracy o1 expectancy 
^ tables-two and thre^ years older than the f^esbman 
grade expectancieis they were predicting/ 



The ACT Assessment Progfam 



The ACT Assessment Program is a comprehensive 
evaluative, guidance/ and placement service for 

' students and educators myalved m the transition of. 
stLKjerits frdm higK school ta college. This program 
IS based on the ACT Assessment, which consists of 
four academic tests;. gQif-reported high school, 
grades, the Student Profile'Section (SPS). and the 

, ACT Interest lnvento.i:y The program gjso includes 
\h number of supporting research servicJes, 
cfescrjbed below.. . . ^ 



When students register for the ACT Assessment 
Program., they report the last grade received in each 
of the four subject areas prior to the senior year of 
high school. The arithmetic average of these four, 
grades— defined as the high school average— pro- 
vides another measure of overall academic ability. 
Maxey and Ofmstjy.' (1971) . ipivestigated the 
accuracy.of self-reported high, school grades and 
founa that about 78% of the students reported their 
grades correctly, 



The four academic tests of ^ha ACT Assessmerit 
measure abilitie.s. m the subject areas traditionally 
identified with college and high s'chool programs: 
English, mathematics, social studies, and natural 
sciences _The^ English Usage Test measures 
students understan^ehng and use of the basic 
elements of correct and effective writing: the 
Mathematics Usage T^st measures their mathe- 
matical reasoning and problem-solving ability: the 
Social ■ Studii^s Reading Test measwres the 
problem-solving skills required in the • social 
studies, the Natural Science Reading Test mea- 
sures the critical reasoning and problem-solving 
skills required in the natural sciences.' The 
arithmetic average of the scores on these four tests 
IS the ACT Composite. score, which is often used as 
a measure of overall academt%: ability. ACT test 
scorps are reported on a standard scale that ranges 
from 1 ' to 36. More detailed descriptive and 
technical information about ACT test scores can be 
found in the Technical Report for the ACT Assess- 
ment Program (1973). 



Another component of the ACT Assessment 
Program is the Student Profile Section (SPS).. 
Through 'the SPS. students provide information 
about ,their background, extracurricular accom- 
plishments, .special acadeniic peeds, ^ housing 
plans, financial peeh; planned major, -and carreer' 
plans. Students complete the SPS "when they, 
register for the ACT Assessn^nt. 



The ACT Interest Inventory rYieasures students^- 
preferences for job-related activities in six basic 
interest dimensions: Science, Creative Arts. Social 
Service, Business Contaiit, Business Detail, and 
Technical. Students complete thejnterest. Inven- 
tory when they^register for the ACT Assessment. 

Information from all these sources— the ACT tests* 
high school gi;ades. SPS. and Interest Inventory— is 
organized into Individual Student Profile Reports 
sent to students and colleges. With the summary 
information provided by the Student Profile Report, 
students and 'educators can. make ^ informed 
decisions and plans. • ' ' . 



The ACT Predictive Research Services 



ACT offers Without- charge to colleges two general 
.plans for predicting freshman grades. Each plan is 

deaigne^to meet the varying needs and resources 
.^f colleges which, use ACT data. 

. The'feasic Research Service requires minimal effort 
by a college in reporting^data. To participate in tine 
Basic Research Service, college personnel, simply 
mark on a computer-generated roster the ove/all 
grade point averages of each of their freshmen. A 
n^inimu.m data base of ICfO records is required. 
Through ^ Basic Research Service, a college can 



obtain ^predictions of overSit freshman grade 
average based on the four AGT test^scores and the ' 
high school average. If they Kave data for at least 
too students of each sex, -colleges can also obtain 
separate prediction . equatrons for 'males and 
females. During the f 977-76 academic year, 461 ' 
colleges reported grades for 126,880 freshmen 
through the Basic Research Service. 

g • . " 

The. standard* Research Serviq^s ' designed fqO. 
colleges . that . want to predict specific course ^ 
.grades, develop prediction equations for sub- 



G 



ERIC 



groups/pf students, ^or use -predictors other than . 
ACT test scores and high school^grades. A large 
variety of research studies can be gccommodaled 
by the Standard Research Service, Data can b.e 



supplied either through punched cards, optically 
scanned cards; or magnetic tape. During 1977'-78, 
185 colleges, reported grades lfor'150,998 students 
through the Standard Research Service. 



Data Base 



*This study IS based on a sannple from a data base 
consisting x)f. student records submitted by institu- 
tions. throcigh their participation in ACT'-s predicTiv^e 
research services. The institutions represented in 
this data base participated in the ACT predictive 
research services in the academic year 1976-77 and 
in one or more of the academic years 1972-73, 
1973'-74, 1974-75. and 1975-76. ■ Therefore, 'the 
grades earned by the 1976-77 freshnnen' at. these 
institutions "were • avarlable for comparison with 
grades predicted from equations developed in one 
or more of the four preceding years. There are 605 
colleges represented in the data base. 

Because the data m the study' were collected from 
colleges participating in ACT's predictive research 
services, in some respects they are ndt representa- 
tive ol students nationally: , 

• Colleges using the AlCT Assessment are located 
-mainly in the* Rocky . Mpuntains, Great'Plains, 

South, and Midwest with comparatively fewer in 
-the East' Northeast and on^e^West Coa^. - 
' . ■ • 

• Privately controlled institutions . are rplatiyely 
underrepresei^ted among colteges that use the 
ACT Assessment, and publicly conkolled instfHj- 
tibns are over represented. 

• Because participation in ACT s f^es'earch services 
IS voluntary, t-he data base is self-selected eyen 
among colleges that usq the ACT Assessment 
Program . 



or 



The results of the . study^ are, therefore, .not 
. necessarily representative of the results that would 
be obtained if data from all colleges in the nation 
could somehow be collected.. One should be 
cautious, therefore, in applying the results to 
institu<ions which. do not use the ACT Assessment 
or do not participate in ACTs predictive research 
services\ Nevertheless, the study will suggest major 
trends and extend. knowledge in this area beyond 
the results available to date. 

To, maximize the numbe^ Of colleges from which 
data were available, the variables used in the 
predictions were restricted to the followmg 
specifications: overall grade average as criterion, 
Nhigh school grades and ACT scores as predictors, 
ajjd subgroupingtQ43 sex. 
. ^ " 

Most of the grade c^veraQes in this study are from 
the first semester of the freshman year. Colleges 
participating in some of ACT's research services do 
havQ the option of pooling grades from j)revious 
years or reporting grade averages based on- the 
entire freshman year. ACT does not maintain 
"records of individual colleges' choice of criteria. 
However, from examining production volumes 
throughout the year, we. estimate that Over 60% of 
the colleges InMhe data base reported first semester 
grades for the curraht academic year, and the rest 
either reported first year cumulative grade averages 
0^ pooled data from previous years. 



Sample Design , 



To reduce the computatior\al costs of this study, 
weights were ci^lculated and prediction equations 
were cross-validated on a probability sample- of 
recrords sd^ected< from the above data base. The 
sampling was carried out in two stages. % 

Firstra random sample of 260 colleges was selected 



from th§ 605 'colleges in the data base. .Weights 
were c?emputed Irom all student recorda^submitted 
by these 260' institution^ in the aeadenguc years 
^1972-73, 1973-74, 1974-75, and 1975-76. Observe 
^that.n^t every institution supplied dati in every 
year. The number of colleges by year* in*the data 
base and sample is displayed in Table U • 



X- TABLE 1 

Summary of Data Base and Sample for Cross-Validation Study 



Year 


Number of^ 
colleges in 
database 


7 7 • Number of 
^ colleaes 
in sample 


1 ' ^ 

Number of student 

rpcord^ for ^ 

computing vi^ights 


Number Of 1976-77 

siuaeni recoras 
for cross-validation 


1972-73 . 


451 


187 


97,985^ 


' ; ■ 13,732 


1973-74 


484 


203 


114,331 


15,589 


1974-75 . 


'•494 


* 

■ _ \ 207 

'\ ' - 


108,118 r 


15,412 


1975-76 


520 


\ ^'211 

I 


109,207 


15,384' 

1 


1976-77 


605 

> 


, , ■ '' 260 


• 


m 




. l - 




■<V ■* ^, r^*^' ■-'"^■".jt . . r..i -^rt 
■ •*■>. ■ 





F'or each college in the sample, a random 
subsample of the 1976-77 student records Was then 
selected. The differences between actuaM1976-77 
grades' and predjctiohs based on 1972-73 equa- 
tions. 1973-'74 equ^ions. and so on. were sum-' 
manned. Thus ^ cross-validation was, made for all of 
the equations developed from data in the four years 

preceding 1976-7V. 
♦ ■*• 

At the time the study was done. 1976-77 grades 
^ were^e most recent available for cross-validating 
predicTibn equations developed from earlier years' 
data. Qorfipu^ational costs prevented repficating 
the stbdy using earlier: base years and cross- 
yalKjation years.y^ / . - 

It' was anticipated that the validity of grade 
predictions might vary according to differeril&es in 
' the. composition arid affiliation of the colleges, By 
taking note of Jhese differences when designing the 
sample. it»was hoped that greater precision in the 
inferences might be obtained. .In this study, 
..separate samples were sefected from strata of 
. colleges determined by their control, type., and size. 
Tbe stratifrcation variables were: 

• The control oi a colfege, pu|)lic or private. 



The type of a college, as determined by the 
highest degree level it offers: 

Type 1: Two but less than four years o1 work 
beyond Grade .12— ihcludes junior colleges', 
technical institutes, normal, schools. 

Type 2: Only the bachelor's or first professional 
degree— inGlu(^es those institutions offering 
courses of study leading to the customary 
. ';&achelq4 of Arts or Bachelor of Science degree 
and all those degrees wljich entitle the possessor 
. to enter the profession indicated. ' / . 

Jypd 3: Master's and/or second professional 
degree— includes those institutions offering the 
customary first graduate degree and any degree 
earned in the same field after the bachelor's br 
first professional degree. This type of institution 
does not offer t^e Doctor of Philosophy or 
equivalent degrees. . • . 

Type 4: Doctor of Philosophy and equivalent 
degrees— includes thosdtoifetitutions which are 
considered .uniyersities.^r^t. . . r 

• The size stratum for a college, as determined by 
*the number of students for which the college 
reported 1976-77^ freshman grades: 



Category 1: 100 stuctents or fewer 

■ . ■ ■ '. ■ , 

^Category 2: IOt-200 students . , 

« 

• Category 3: 201-500 Students 
. Ca/egory 4; 501-1000 students 

* 

Category 5: 1001+ students 

i 

Colleges do rtave'the option of reporting grades for 
, a sample oi \he\r freshman class, pr^ovided the. 
sample size is 10Q or larger. Moreover, colleges can 
poo}- data from previous years in order to^attain the 
required minimum sample size.. Therefore, the size 
measure in this stutJy is not necessarify the same as 
total freshman class size. However, for about 70% 
of the institutions, the n umber of records^n the data 
base is within 30% of the estimated freshman 
enrollment. ^ 

The subsamplmg rates for 19^6-77 records were a 
'function of the size strata for the college sample. 



Thes'e subsampling rates and ttie number of. 
colleges selected from each size stratcirti' are 
displayed in Table 2. . . 

These 'sample sizes were chosen, to yield a 95% 
chance that the mean absolute 6rror (defined 
below) estimated from the sample would be within 
the following limits of the mean absolute error 
computed. fropi all records Jn thQ data base: 

• .01 grade units of the mean absolute error of all 
students in aircolleges. ^. * 

• ± .03 grade units of the mean absolute error for 
all students in each size stratum, - ' 

The second condition was imp.oeed to permit the 
use of the sample to Study ttieneffecte of college size 
on predictive accuracy (results not reported here). 
Examination of 4he sampling variances Estimated 
from the data indicated that these precisions were 
attained. 



TABLE 2^^ 

r 



stratification and Subsampling for the Cross-Validation Study Sample 

by Size Stratum, ^ " , 



Size . 
stratum 



Number of colleges 
in data base 



100 or fewer 
101-200 
201-500 
501-1000 
Over 1000 



47 

r 

22 L 
202 
70 
65 



Number of colleges 
in sample 



38 
75 • 
63 
44 

« 

40 , 



Subsampling ra^e . 
for 1976-77 records 



1/2 
.1/4 
1/4 
1/6 
1/16 



Total 



605 , 



268. 



Prediction Equations. 



Prediction equations were calculated from the 
1972-73. 1973-74. r974^75, and 1975-76 data using 
a standard four-variable multiple linear regression^ 
Of college frp^hman grade average on the four ACT 
t#st' scores: ' 

■>-a, ; ACT English score 

♦ ' • ■ 

• ACT Mathematics. score 

' • ( ■ 

.< *-a^ * ACT Social Sttjdies .score \ \ j 

+64 * ACT Natural Science scdr^' ' 

where a,), a,, a^, a^, and 84 are regressiori weights 
calcu^ated from the J^ase year data. In the ACT 
Standard Research Service this prediction is called 
ttie T Index. 

Prediction equalionj^ere also calculated using': 

• the four self-iteported high school grades only m 
a multiple fin^ regression equation (H Index). 



equation (f 

• the averag^.of the T Index and H Index (the TH 
Index) 



the four.ACT test scores and the four high school 
-grades in an ei^ht-variable multiple ' linear 
regression equation. 

Past research has sh^wn vftiat the correlation' 
betweenjhe TH Index and freshman college graqjfe 
average ^s only slightly less than tbat. for the 
^traditional eight-variable multiple linear regression 
equation (Technical Report for the ACT Assess^ 
ment Program^ 1973). , 

•-'0 ' ' ' ■ ■ ' - •' 

Previous research also suggests that the relation-" 
ship between high school grades, standardized test 
scores/ and freshman grade, average- differs few 
males and females (79!cA7n/ca/ Report for the ACT 
Assessment Prograrrj, H973). Moreover. coHeg^ 
ofteri elect to receive separate predictions'by sex. 
, For these reasons, prediction equations were 
calculated separately for mal4s and females in each 
college, asr well^as for all students in a college. 

Some colleges in the sample submitted student 
grade data without identifying the sex of their 
studeniS/ Therefore, the sum of the rrumbef, of 
, male's and females from which the separate-sex 
equations were computed is less than the total 
number of records from which the combined-sex 
equations were computed. 



t:ross-VaHclation Statistics 



The difference between 'the predicted and actual 
1976-77 grade average for a student is called th6 
error of prediction. Thj^ magnitude of this errOr. 
ignoring 4he direction, is called the absolute error 
of prediction If the absolute error of prediction is 
averagedover a group of stddents (for exampj^. all 
the students in.a college), [he resulting nuTriber is 
called the moan absolute ertor (MAE) of prediction 
in that group 6\ studen;3. This ^statistic has an 
immediate relevance to ,tne. quality of grad^ 
predictions. For example, if the l\iy\.E in predicting 
grade average in a college is .45. then, on the 
average, there is an absolute discrepancy of .45 
grade umts between the predicted and actual grade 
averages in the college, ^ 

A related measure of pjediction quality Jhat is 
sometimes used is the mean squared error, which is 
the average over a group of students of the Squared 



error tff predict>6n, .It is-actually this quantity that 
, the usual least-squares regressior^/mtetoods seek to 
minimize- Because the mean squared error is leas 
directly interprelable in practical terms than the 
MA&, it' was not used here. 



Another^useful measure of the quality of prediction 
for a group of students is the proportion whose, 
predicted grade 1^ within a certain 'range of the^ 
actual grade. In this study we report the proportion 
of students whose predicted grade average is 
withln"O.20 grade units o| actual gradg average; this 
•proportion will be denoted by the abbreviation P2p. 
An absolute error of 0.20 prade units represents a 
high degree of accuracy in prediction. The statistic 
P20. therefore, measures the proportion of students 
fpr whom extremely accurate grade 'predicllons 
Were possible. Some researchers'* may con^der 
prediction* errors larger than 0.20 to be ^quite^' 



acceptable.' analogous proportions (such as the 
proportion of students whose predicted . grade 
average" IS* within 0.50 grade units' of the actijat^ 
grade point average) could be computed for therh. . 

Probably the most commonly used-- pleasure' of 
predictive, efficiency* IS cross-vaticJated r, that is, 
the Pearson correlation between the predicted 
and' actual gf'aQfes. , Generally, this, coefficient 
IS compared .with, the correlation coefficient 
calculated from the base ye5r data to give an indi- 
Cation, of the aJT^uracy and stability of the . 
prediction equi^ions. Because it is so wi.defy used, 
th'fs statistic IS refjorted.in addition to MAE and P20. 

A limitation of 'this statistic is illustrated by the 
hypothetical ^ta in Figures la and lb. These plots 
sugge^f thatMalafe conclusions could result from 
using cross-.validated- r as the sole measure of 
prediction quality In both cases, cijoss-validated r 



, IS near 1, yet in both, cases the prediction Is not-very 
good, in the fO-s't instance^ the predicted grades are 
consistently * too low; in the second, they are- 
systematically too low for low-ranking students and- 
too high tor high-ranki-ng students. Thus, a^ large* 
cross-'validated r does not guarantee success in> 
. prediction. A low value otcross-validated r wilj i ndi- 
cate poor prediction, , however, because ". if the 

• predicted arid * actual ' grades have little- linear 
relationship, -they 'cannot be very close, rtp each 
other. * ; ■ ' ' - 

A justific^ion ^ sometimes .'advanced for the 
/correlation! coefficient is .that r measures the 
pfopoYtional decrease in squared error from that 
wtiich '\vould result if the mearr grade were 
predicted^or each student. Similarly, jakes into 
account the effect on prediction due to the inherent 
variability'of students* grades. 
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gure 1. Two situations in which cross-^ahdated r gives a falsejndication of /prediction quality^; 



Compiitations 



The . prediction equations and' Gross-validatipn 
statistics tor ^ach college Were computed hy a 
custom program using the IMSL subroutine 
package (IMSL 1977). The'program was tested on . 
data from three colleges- over a three-year. period. 
Every computation on these^data was , compared 
with one obtained from the SPSS ►statistical 
package (Nie," Half.^ Jenkms, Slembrenner, aad 
Brent. 1975r and, where feasible, with one^ 



f 

In 



,com|>uted by hand. In all cases, the cross- 
validation statistics, for e^crt college we're within 

, ± ,001 of each otfier/and in a majority of.cSses they 

' were wjthin ± ,0001, ^ * - 
• " * *. ' 

The cross-validation statistics from each college 
were, summarized through Sf?SS routines, Theflata- 

' were weighted to take inb account the differential 

• sampling rates in the various strata. 
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Results 



The results o( the analyses are displayed in Tables 
3a-3d. 4a-4tl. ^ and 6. They indicate that for 
.students ^as a w'hole (irrespective of'thei/ college),* 
the accuracy of*^ all four kinds of predictien 
equations- was quite stable over the four-year 
^perio^ studied Thte same is generally true of the 
accuracy of the prediction equations for colleges. 
although a few' inclividuaUcolleges do show some 
wistability 

Tables 3a-3d display tlie cross-validation statistics 
•for all students iry.the sampl0. Over the four year 



period, there is no rparked increase or decrease in^ 
any of the three statistics for the four kinds^f 
predictions. For the T, Index, MAE for the 'tot^l 
group of students is about ,57, P20 is ,22zj^, and.r 
iS. .48-,49, For the H Index, there is slightly more 
deterioration over time: t^AE. ranges* from ,57-,60, 
T20 fr,om ,21-.23. and rfrom .,49-,51, Jhe VhUndex 
predictions ^ave an MAE of ,54- 55, a P20 of :23-,24, / 
and an r of .55-, 56, The eightrvariable regressiorr 
equations ^hrpwed virtually the same results: ^MAE 
from ,53-.55, P20 from .24- 25, an^l r from .55-:56, . 



TABLE 3a 



9- 



Accuracy of Prediction of 1976-77 student Grade Average 
from Equations Developed in Years 1w2-73 through 1975-76 

(Tlndex— ACT Test Scores Only) 



Grpup 

Tota^ group 



Cross-validation 
statistic 



Year In which, prediction equation was developed 

1972-73 . 1^73-74 1974-75 1975-76 



Malqs . 



P20 

^ ■ r, ■ 

MAE 

; P20 



Females 



1 ; MAE 
P20 
r 



.57 


'.57 


.57 


' .57 


.23 . • 


.22 


.23 


.23 


.48 • 


.48 ■ 


.49 




M 


.61 ' 




.61 


.21 • 


.20' ' 


/ .21. . . 


.21 


.43 


.43 


.43 


.43 


to 








.56 


.55 . 


.. . - .55 


.55^ 


.22 




.25 

» 


'23 


.53 


• .50 


.51 


* .51, 



^ /% TABLE 3b 



Ateijracy of Predictiort of 1976-77 Student Grade Average 
from Eciuatibns' DeyelofWd in Veara t972-73 through J9^S-76 

\ ' (H fndex— High Schpol.QTades Only) . " ^ 



Group ! 



Cross^v^alidation 
statistic 

i 



Year in which iye<jiction eq(Uatron; was enveloped ^ 



1972-73 



JL 



197fl^7^ 



1975-76 



Total group. 



. Males ft \ I 



,5^ FemaJes 



• Ik 



MAE y 
r 

a 

P20 
r 

MAE 
P20 

r . : 



'.f 



.49 

.62 

.46 

.57 
.24 
.50 



.59 
.23 

^20 
.46 

.55' 

.23 

.53 



•57 

.22. 
.50 

.,.60 
.22 
. 46." 

"..54 
.24 
.54^ 



I? 



'. .57' 
. ^ .23 



.51 

.60- 
.21 



.46 



* 24 

.53 



(9 
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TABLE*3c 



Accuracy of Pirediction of IS/e-^r Student Grade Average 
froin^Etjuaflohs Develo))ed in Years 1972-73 through 1975-76 

■ ' ^ ... ' > " (TH Index) 




«i-validation 
tatistic ' 



Year in which prediction equation s developed 



1972-73 



1073-74 



'1974-75 



1975-76 













1 f 






TojaK group 


V. ■ < .-maeX 




.55 




V 


54 


^54• 




• . • '^20 \ 






23 




,24 


.24 V 

• 




* ■ ■■ r / \ 




.55 ■ 


.55. 




.56 . ■ 


.56 


MaJes . 


MAE 




.59 


.59 






.58 




^ P20 






.22 




.23 


;23 


-> 






" .50 


.50 




.50 • 


.51 


RamalGs ^ 


- MAE.' 




.53 


' .52. 




.52 

* 


.52 ' 


o - •• 


. P20 ' . 




.24 


, .24 




.26 


.'26 


'■■ . ' 






.53 


.58 




.59 


.59 ^ 






'If- 
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Accuracy of Prediction of 1976-77 Studenf Grade Average 
froffi Equations Developed In* Years 1972-^3 through 1975-76 ■ f 

(Eight-Variable Multiple Unear Regression) * 



Cross-validalion 
statistic 



Year in w^ich prediction equation w^as developed 

1972-73 1973-74 . 1974-75 • 



1975-7 

4 











1 






• — — 


Total ^roup 


% 


MAE 




.55 ' . 


• 


.53 


.54 




\< 


P20 
. r 




. ,24 . 

:55'»^' 


.24 - , 
.55 


.. ,25 
.56 


.24 

, .56 


Males 




. ^.^^ 




.59 


\. :59 


♦ ' .58 . 


.58 . 


<» ■ 

r 




P20 ^ 

- r 




.22 . 
."50 


.23 
* .50 


' 23 
.50' 


\*23-^ 

1 

.51 ♦ 


Females 




JVIAE^ ' 




, .53 


.52 


.51 


.51 . 






P20» 




.24 


,..25" 


.26 


.25 










.57 


. . -57 


.59 


.69 



The J-x Index predictrons were somewhat more 
stable over time than the H ir>dex predictions. Tlie 
reasons for this cannot, be easily determined, "but 
could be related to ..changes in high schoolt 
curriculum or inflation I'rr high school grades. ' 

Although the H Index r's are slightly but constantly 
larger tl^an the T Index r's. tire statistics MAE and 
P2Q mdiqafe that, on^ the average, the T Index 
?esulted in slightly better predictions than did. the H 
Index.. That the H Index r is ^arger tha/i the T Ifidex r 
IS consistent 'with earlier publtshed Research (ACT 
Technical Report 1973). The.rjBsults for MAE and 
P20, however, suggest that something like the 
"situation illustrated in Figures 1a and ,1b occurred 
in many of Jhe colleges. 

Further exarnin^tion of*^ the prediction "^^errors 
revealed that the H Index predictions based on the 
1972-73,* 1 973-74. 1974-75, an/j 1975-76 data under- ' 
esiimated the cjriterio'n on the -average by .14, '.10, 
.04,^nd less than .01, respectively. The T Index 
predictions based on data f^^om these four years 
often overestimated thacriterio/i, but by an ayerefge 
of only .05, .05, :05,'and .03, respectively. Thus, the 
H Index tended to underestimate th^ criterion,'and 



the amount of Underestimation increased with the 
age of the prediction equation. The T Index tended 
to overestimate the criterion, but by a fairly 
constant and small amount over time. 

Prediction equations developed for males and 
females separately show^ the same stability in 
quality as the total group equations.' The 
predictions for females were somewhat more 
accurate (eight-vanable multiple regres$ion MAE = 
.51-.53, P20 = .24^.26, r ^ .57-.59) than the predic- 
tioris for males (eight-variable multiple regression 
MAE = .58-.5*9, P20 = .22-.23, r = .50-,51), This result 
i,s cOnsi;5tent- with earlier analyses of ACT grade 
predictions (ACT Technical Report, 1973), 

It should be noted that'these results pertain to 
separate-sex predictiop equations, rather than to a 
total group equation. The cress-validation results 
/ for predictions made from* combined-sex equa- 
,'^tions, however, are virtually identical Jo those 
for predictions ' made from the separate-sex 
equations. This would indicate thaf there is 'little 
average tjienefit in developing separate-sex 
equations. V 



Tables 4a-4d display the college medians of tlie ~ . measures, of predlptlon quality for individual* 

cross-yaffdaiion statistics. The numbers in this colleges. The sameTiitabilKy in ojpdiction quaii.|y 

table show that m^easures of prgdiction quality for ' over the' four year's is evident;' , 
studenj[s across all cQlleges are also'' typical .of the ' > ■ , 

» TABLE 4a ^ . 

, Wedian College CrbssrValidation Statistics for Prediction Equations, 
I Developed from Data in Yeqrs 1972-73 through 1975-76 ' . ^ 

(T Index— ACT Test Scores Only) • . ' 



Group 

Total group 



Males 



Females 



r^rn^Q-valiHAtion 

. statistic 


Ye'ar in which prediction equation was developed 


1972-73 


1973-74 


1974-75 


1975-76 












MAE 


.55 .. ; 


.56 


.55 


' .55 


P20 


. .23 


. .23 


.22' 


.22 


f 


.48. 


.48 


.48 _ • 


48 


' . MAE . 


.60 


.61 


.60 


.60 


P20 


.22 • 


.24- 


V ..20 ■ 


.20 


r . 


.44 ■ 


.42 


.44 


.42 

• 


' MAE . 


.54 


; .54 


.53 


.54 


♦ P20 / 


.23 


.24 


.24 


.23 


r 


.50 


.52 


.53 


.51 



f ■ ' • TABLE 4b 

Median College Cross-Validation Statistic^^ for Prediction Equations 
Developed from Data in Years 1972-73 through 1975-76 

(H Index— High%chool Grades 9nly) 



r • T-r- - 

Qroup 


Cross-yalidation 
; / statistic 


Year in which prediction equation was developed 

1972-73 '•1973-74 -1974-75 1975-76 


Total group 


MAE 


.58- 


.57 


.60 


.54 




P20 


■ . .21 


■ V^^ . 


.23 


.23 




r . 


*. .48 


.30 


.49 


.50 


Males ' C 


MAE 


.60 ' 




.59 


.59 




P20 • 

r 


.20 


.20 


.21 ■ 


^22 






.45 


.47 . 


.45 


.48 


Females . 


MAE 


• .57 


.54 


.53 


.53 




. .P20 


. 2.1 . . 


-.22 


.24 . 


.23 


V 


r 


.54 


?.53 


,.54 


.55 



V. 



■ • y ■ .■ ' \ TABLE 4c ^ ' 

hd\6n College Cross-Validation Statistics for Prediction .Equations 
h pe^elope(t,from Data In Vears 1972-73 through 1975-76 

'' * v; ,. (TH Index) 'l' ■ ■ • 



% — f- 



. CMsi-yajidalioiv 
' statistic ■ 



Year in whlci-^ prediction equation was deveJoped 

1972-73 , .1973-74 1974-75 .1975-76 



TptaiVgroup 



Males 



Females 





•i < 



















MAE 


.53 


.54 


,52 


.,52 


P20 


. .24 




f. -24 ■ 


. ' .24 


r ■ ■ ;■' 


-.55. 


.56 




lyiAE 


- ' .56 


,59 


. .56 


.56. 






.21 


- ■ .23 , 


. .23 






.54' 


..53 


.^5 


MAE 


• .52 


.51 


.50 


.50 


P20 


.24 


,24 


.25 


.24 


r 


.5S ' 


■ ' .,60 


:60 


.59 
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TABLE 4d 



Median College Cross-Validation Statistics for Prf diction Equations 
Developed from ^pita in Years 1972-;73 through 1975-76 

(Eight-Variable" Multiple Linear Regression) 



Group 

Total group " 



Males 



Femrales 



Cross-validation, 
statistic 



MAE 

P20 

■r 

MAE 
P20 . 
r . 

MAE 
.P20 
r 



Year in whiqh prediction eqiiatjon was developed 

1972-Z3 1973-74 1974-75 -1975-76 

■ — V — 



.53 

■24' 

.54 

.57 
.21 
.52 

.52 
.25. 
.57 



.53 
.24 
.55 

.60 
.21 
.51 

.51 
.25- 

!57 



.52 
.25 
.56 

.57 
.21 
.51 

.50 
.26 
.59 



.51 
.25 
.56 

1 

!56 
.23 
.51 

.52 
J25 
,68 



-f- 



Table 5 presents' a frequency distribution of MAE 
for the tfight-varjable regression. among coljeges 
for the four years^tudied. For about 72%-78% of the 

..colleges^e MAE Is 0.60'or less. Thus, the MAE 
reported in Table 3d for stu'dents over all coHeges is 
fairly typical of the MAEs in most colleges indi- 
vidually. For a ^small /pToporJion «(2%-4%) of 

.colleges, MAEs ip the range 0.80-1.10 occurred. 

The reLativ;e stability of the quality of predictions 
over all students and colleges could mask 
instability in particular colleges, Table 6, wliich 
addresses this issue, is a freguen'cy distribution of 
the 'differences between MAE for eight-variable 
.multiple regression^equationsdevelopecl^in 1975-76' 
and MAE for the equations developed in 1974-:75, 



' 1973-74, and 1972-73.,The results indicate that for 
; most colleges,- MAE is quite stable irom' yea i/'to 
^year. The number of coll^gefi vvith SNargefr /lAE 
? from 1974-75' prediction equatiOQs than fr.orn WS- 
76 prediction equations is approximately the same 
■ as the number of colleges with a smaller mAe from- 
1974-75 prediction equations than fr.onr) 1.975'76 
equations. Moreover, about 99%. of the ^iffere/'nces' 
rnfMAE are in the range -.10 to +.10, Sli)g[lfly more 
colleges^ have a larger MAE f/orn' equations 
develppecf in 1974-75, 1973-74, Or .19.72-73 than 
197.5-76, but .tfce' differences in MA,^ are concen- 
trated in the range, of -.10 to .Op. ;n summary, .the 
MAE for over 90% of the col lege?.' remains stable, 
even over a period of four ye$rs. • 



tabCe 5 • . * 

Proportion of Colleges with Varlo.us Ranges of Mean Absolute Error 
in Predicting 1976-77 Freshman Grade Point Averagie 

(Eight-Variable Multiple Linear Regression) 



Range in MAE 



Year in which prediction equation wiis developed, 

' — ' '• \ — \ 

^,1972-73 19^3-74 1974-79 ' 1975-76 



0.0 r 0.1 
0 .l(- 0 2 
0.2 - 0.3 
0.3 - 0.4 



0.4 

0.5* 

0.6 

0.7 

0 8 

0.9 

1.0 



0^- 
06 
0 7 
0.8 
0 9 
1.0 
1.1 



.00 


■ .00 


.00 




.00 


.00 


' ■ .00 






.00 


.00 


.00. 






.00 


.07 




.12 






.31 


.31 . 


-.29 






.34 


•34 


• .37 




.33 ' 


.18 


.21 


.15 




.13 ' 


,09 


.05 


.06 




.05 


92 


.02 


.01 




.02 ' 


.00 


. .00 


.00 




.01 • 


.00 


•00 






.01 



f Sfafoility of Mean Absolute Error across Four Yearsr ^ ' / 
' Proportions 6f Colleges with Various Difterences in MAE from 7 J 
^975-76 iqujitlons and MAE from Equations Developed in Earlier Veirs. '^jtf^^ - 

(Eight-ya'riable Multiple Linear Regression) • Vt' 



ange of difference in IliAE 
from 1975-76 ^quationeahxl 
AE from older equa«|t»n 



Earlier year in which prediction was developed 

>972-73 1973-74:- h ' M&74r7§ 



-0.50 to -0.40 




■ t • 


. .00 


-0.40 to -0.30 






.00 . 


-0;30 to -0.20 






;00 


-9.20 to -0.10 


\\:. 




.06 


-a. 10 to 0.00 


■U- ' 




.63 


0.00 to 1.10 






.30 


0 10 to 0.20 




.00 


0 20 to '0.30 






.• OK./, 

.00 


0.30 to 0.40 






0.4.0 to 0.50 






.00 


0.50 to '^.60 






.00 



.00 
.00 
.00 
.03 
.69 
.28 
.00 
.00 
.01 
.00 
.00 



.0^ 
OCh 
.00 
.01 
.55 
.44 
.00 . 
.01 
.00 
.00 . 
\00 



' ^ ■ • - • 

7" " . Sun^mary and 

* ■ . " ■ t - ■ ■ 

The s<al3il»iff of predictien equations based on ACT 
scores aj^i high school grades was investigated for * 
9 strati1i&&", randorn sample of 260 colleges that 
participated in the ACT Research Services during, 
the p^i^od of 1972-73 to 1976-77. Separate 
predictK5n equations [or. each college wer^ 
calcul^El ff<?rTi dat9forthe.year§ 1972-73, 1973-74; 
•1974-?6> and '"1975-70 and applied to' 197^-77 
freshij^. Prediction errors for. 1976-77 grade • 
ayeradi^ wer« then summarized. ' 

The results inditate that for most gtudents' and 
colle0|ps, the quality of the prediction equations is 
quite^stable over the four: years. The mean absolute ' 
error;|t|^^^^ using an eight-variable multiple' 

regre^s&lon equation ranged from .53 toi.5^; the pro- > 
portion of students whose pretiicted grade was 
with^^^O of their earnejj grade Vtinged frtfm .24 to 
.25; %nd the cross-validated correlation ranged 
frorTi?^55 ta:56. A similar stability wds noted in the 
accur^by of predictions for males ancf females 
usia»*W)arate-sex prediction equations/' 

Ictions based on test sporei only had. a 
snrialler mean absolute error than grade 




Conclusions . . ' 

predictions bas'^l^^ high school grades only and 
^showed slightly greater stability over time, 'The 
mean. absolute error of predictions based on test 
scores only was about .57 for the whole' four year 
period.. For (Dredictions based on high school 
grades only, it ranged frp/ri ^57 .to .60 over the four 
year period. 

One can conclude from this study that although 
most coHeges experience some change in th^ 
students' academic atylities. their curricula, or 
grading practices over^a. period of time, freshman 
flr^tle average can in'-m^DSt cases be. accurately 
^edicted using equations which have been in use, 
for as long as four, years. It is^ACTs policy that 
colleges update their prediction equations at least 
every three years if they are receive grade 
predictions for future applicants. The data from this 
study suggest that more-^frequent Vevision is not 
generally necessary. In considering " whethei' 
revision of equations is rtecessary before the 
required date, college researchers should, of 
course, determine whether ^ some change vrhas 
occurred which would necessitate. an early, revision. 



f , 
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